Stochastic Games on a Product State Space

Stochastic games and product-games. An n-player stochastic game is given by (1) a set of players N = 1 n , (2) a nonempty and finite set of states S, (3) for each state s ∈ S, a nonempty and finite set of actions As for each player i, (4) for each state s ∈ S and each joint action as ∈×i∈N As , a payoff r i s as ∈ to each player i, (5) for each state s ∈ S and each joint action as ∈×i∈N As , a transition probability distribution psas = psas t t∈S . The game is to be played at stages in in the following way. Play starts at stage 1 in an initial state, say in state s1 ∈ S. In s1, each player i ∈ N has to choose an action a1 from his action set Ais1 . These choices have to be made independently. The chosen joint action a1 = a1 a1 induces an immediate payoff r i s1 a1 to each player i. Next, play moves to a new state according to the transition probability distribution ps1a1 , say to state s2 ∈ S. At stage 2, a new action a2 ∈As2 has to be chosen by each player i in state s2. Then, given action combination a2 = a2 a2 , player i receives payoff r i s2 a2 and the play moves to some state s3 according to the transition probability distribution ps2a2 , and so on. We assume complete information (i.e., the players know all the data of the stochastic game), full monitoring (i.e., the players observe the present state and the actions chosen by all the players), and perfect recall (i.e., the players remember all previous states and actions). A Markov transition structure i for player i ∈ N is given by (1) a nonempty and finite state space S; (2) a nonempty and finite action set A si for each state s ∈ S; (3) a transition probability distribution p siai si over the state space S for each state s ∈ S and for each action a si ∈ A si . Note that, if we also assigned a payoff in every state to every action, then we would obtain the well-known model of Markov decision problems for player i. We will now consider a special type of n-player stochastic games, called product-games, in which the transition structure is derived by taking the product of n Markov transition structures. A product-game G, associated to the Markov transition structures 1 2 , is an n-player stochastic game for which (1) the set of players is N = 1 n (2) the state space is S = S1×· · ·×Sn (3) the action set for each player i ∈N in each state s = s1 s ∈ S is As =Asi ; (4) the transition probability distribution psas , for each state s = s1 s ∈ S and for each joint action as = as as ∈×i∈N As , is psas s̄ = ∏ i∈N p sas s̄ i for state s̄ = s̄1 s̄ ∈ S. Note that there is no condition imposed on the payoff structure. Observe that (1) the action space of player i only depends on the ith coordinate of the state, (2) the ith coordinate of the transitions from any state s only depend on the ith coordinate s of the state and on the action as chosen by player i; i.e. for any s̄ i ∈ S we have psas S 1 Si−1 s̄ Si+1 S = p sas s̄